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Abstract 

CES4Health.info was launched in November 2009 as an 
online mechanism for peer reviewing and disseminating prod¬ 
ucts of community-engaged scholarship in forms other than 
journal articles. One year after its launch, the authors conducted 
an online survey of CES4Health.info contributing authors, 
reviewers, and users of published products. Early evidence 
suggests that CES4Health.info may be helpful for recognizing 
community-engaged scholarship in promotion and tenure pro¬ 
cesses and for providing communities with resources to address 
community health concerns. 

Setting the Context 

D r. Richards (an assistant professor of public health at a 
research-intensive university), Arts Force (a youth arts 
organization), and AIDS Aware! (a community-based 
HIV/AIDS awareness and advocacy organization) developed a 
reciprocal, respectful partnership over several years. Together, they 
designed and conducted a rigorous mixed-methods investigation 
of the attitudes of young people in their community toward sexual 
risk-taking behavior. One product of this collaborative research 
endeavor was a video aimed at high school students, produced by 
the youth with input and guidance from the adult partners. The 
edgy, hard-hitting video communicated information about social, 
emotional, and health outcomes of sexual risk-taking behavior 
based on Dr. Richards’ review of the literature and prevention mes¬ 
sages informed by the findings of their collaborative research. The 
video was integrated into health classes in area high schools and 
utilized by the local health department in community-based health 
promotion initiatives. 

Although fictitious, this is an example of an innovative 
product of rigorous community-engaged scholarship created by 
academic and community partners. Similarly, a service-learning 
partnership could author a policy report of options for eliminating 
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homelessness, or community-campus partners could develop a 
toolkit from the evidence-based service delivery program they 
designed. 

The creators of such innovative products of community- 
engaged scholarship face common challenges (Calleson, Jordan, & 

Seifer, 2005). They lack mecha¬ 
nisms for broad dissemination 
of their work, which often limits 
the impact of their products to 
the local community (Cashman 
et al., 2008; Wolff & Maurana, 
2001 ). They lack peer-reviewed 
publication outlets designed 
specifically for diverse schol¬ 
arly products (Commission on 
Community-Engaged Scholarship in 
the Health Professions, 2005). The 
format of peer-reviewed jour¬ 
nals is inappropriate for such 
diverse products. Moreover, the peer review process may not 
find value in them (Nyden, 2003). Journals may utilize academic 
reviewers that cannot critique the rigor and quality of the engaged 
approach (Ellison & Eatman, 2008). However, without mechanisms 
for peer review and broad dissemination, these products typically 
do not “count” in university faculty promotion and tenure systems. 
Historically, promotion and tenure processes value peer-reviewed 
manuscripts that are published in top-ranked disciplinary journals 
(ASA Task Force 2005; Ellison & Eatman 2008; Gelmon & Agre-Kippenhan, 
2002; Jackson, Schwartz, & Andree, 2008; O’Meara & Edgerton 2005). 

CES4Health.info (http://www.ces4health.info) was developed 
to fill this gap. CES4Health.info is a web-based project developed 
as part of the Faculty for the Engaged Campus (FEC) initiative, 
a program coordinated by Community-Campus Partnerships for 
Health (CCPH), and funded by a 2007-2010 U.S. Department of 
Education Fund for the Improvement of Postsecondary Education 
(FIPSE) grant. 

The purpose of the Faculty for the Engaged Campus initiative 
was to strengthen community-engaged academic career paths by 

1. developing innovative competency-based models of 
faculty development, 

2. facilitating peer review and dissemination of commu¬ 
nity-engaged scholarship products, and 


“[Wjithout mechanisms 
for peer review and 
broad dissemination, 
these products 
typically do not 
‘count’ in university 
faculty promotion 
and tenure systems.” 
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3. supporting faculty in the promotion and tenure pro¬ 
cess (CCPH, 2011). 

In this article the authors describe how CES4Health.info 
works. They also report on the first year of CES4Health.info from 
the perspective of authors, the peer-reviewers of submissions, and 
users of engagement products published on the site. 

Overview of CES4Health.info 

Launched in November 2009, CES4Health.info is a system 
for the peer review and online publication and dissemination of 
diverse products of health-related community-engaged scholar¬ 
ship. CES4Health.info publishes new products as well as products 
that have been previously self-published. Its goal is to publish and 
disseminate results of community-engaged projects through for¬ 
mats such as photovoice exhibits, policy briefs, educational videos, 
and podcasts (Bordeaux et al, 2007). Examples include a policy brief 
about the growing aging population in Chicago (George et al, 2009) 
and a video documenting concern about the health impacts of the 
built environment in post-Katrina New Orleans (Catalani et al, 
2009). 

Community-engaged scholarship can also lead to the devel¬ 
opment of tools to assist other community-engaged scholars (e.g., 
assessment instruments, instructional manuals, patient education 
materials). For example, CES4Health.info published a toolkit to 
establish and sustain a year-long walking program in rural commu¬ 
nities (Zendell & Riley-Jacome, 2009), and a web-based tool to create 
customized family health history materials for families, organiza¬ 
tions, or communities (Edelson, O’Leary, & Terry, 2010). 

CES4Health.info defines “health” broadly to include physical 
and mental health, health promotion, community health, social 
determinants of health (e.g., literacy, employment, food security), 
workforce issues, and professional development. For example, 
Taylor and Maddocks (2009) produced proceedings from a confer¬ 
ence on mental health, delinquency, and criminal activity. Montoya 
(2009) developed a video to make the case that social factors of ineq¬ 
uity make people ill. Jorge and Wilhite (2009) developed a training 
video to help physical rehabilitation professionals maximize their 
clinical interventions with individuals working in agriculture as 
well as to illustrate the clinical case management of farmers and 
ranchers with disabilities. 
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The Development of CES4Health.info 

CES4Health.info was developed in 2008-2009 by a six- 
person design team of community-engaged academics, leaders of 
community organizations with experience in community-aca¬ 
demic partnerships, and individuals responsible for editing 
journals or online resource repositories that publish diverse forms 
of scholarship. The team studied the experience of these journals 
and repositories to develop and pilot review criteria, a reviewer 
application, author instructions, and an accompanying applica¬ 
tion (Reynolds & Candler, 2008; Tandon et al, 2007). In the pilot phase, 
authors were invited to submit products. Peer reviewers were 
recruited and trained. Twelve products were published as a result 
of this pilot phase. Concurrently, a web design firm developed 
and beta tested www.CES4Health.info, including the user inter¬ 
face and the administration tool that permits online submission 
and review of products and management of the editorial and peer 
review process. 

The CES4Health.info Submission Process 

The CES4Health.info submission includes two parts: the 
product itself, and an application. The application records the 
product aims, the quality of the product, the alignment of the prod¬ 
uct’s content and format with the needs of the intended audience, 
and the significance or impact of the product. The author is also 
asked to 

• articulate the scholarly basis of the product 

• describe the community-engaged activities that 
resulted in the product, 

• reflect on the strengths and weaknesses of the 
submission, 

• describe the quality of the community-academic 
partnership, 

• describe the ways that community engagement shaped 
and enhanced the work, and 

• document the extent and appropriateness of the 
engagement process. 

The application becomes part of the published package and is 
considered part of the scholarly product. The combination of sub¬ 
mission of the product itself and the application results in a peer 
review similar in rigor to one for journal manuscripts. 
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The CES4Health.info Peer Review Process 

The review process mirrors typical journal and editorial pro¬ 
cesses. All products are reviewed by four reviewers. The editor 
makes the final determination and communicates the decision to 
the author. A CES4Health.info review also includes some inno¬ 
vative enhancements. All products are assigned to two academic 
and two community-based reviewers. Reviewers participate in a 
one-hour phone training conducted by the editor to ensure that all 
reviewers are well prepared to undertake the review process and 
understand the review criteria. 

The review criteria are well articulated and benchmarked using 
a formal reviewer rating form. Criteria focus on clear goals and 
intended audience, scholarly basis, methodological rigor, signifi¬ 
cance and impact, quality of the community-engaged approach, 
and ethical behavior. The criteria are modifications of criteria artic¬ 
ulated by Glassick, Huber, and Maeroff ( 1997), who operationalized 
Boyers (1990) expanded definition of scholarship, recommenda¬ 
tions from a national commission convened by CCPH ( Commission 
on Community-Engaged Scholarship in the Health Professions, 2005), and 
previous work of Jordan et al., who developed standards of quality 
community-engaged scholarship (2009). The review criteria are 
available at http://www.ces4health.info/reviewer/peer-review- 
process.aspx. 

Since the launch of the online system in November 2009, 25 
products have been submitted (not including products submitted 
as part of the pilot phase); 27% of those products were rejected 
(including some that were deemed by the editor to be inappro¬ 
priate for CES4Health.info and thus not assigned to reviewers), 
and 73% were accepted with revision (all but a few were revised as 
suggested and have since been accepted). All accepted submissions 
required some degree of revision to the product or the accompa¬ 
nying application. 

Features of CES4Health.info to Assist Promotion 
and Tenure Processes 

CES4Health.info includes a number of features intended to 
ensure that published products are favorably considered in pro¬ 
motion and tenure processes. First, authors are provided with a 
citation for their published product that can be included in the 
peer-reviewed publication section of their curriculum vitae and 
in their promotion and tenure dossiers. Second, the number 
of times a product is downloaded is tracked, and is provided to 
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the author upon request. Third, users who download products 
are surveyed. Aggregate information about the perceived utility 
of an author’s products can be provided to authors desiring 
to demonstrate broad dissemination effectiveness of their 
community-engaged scholarship products. Fourth, in the product 
application, authors are invited to provide the names and e-mail 
addresses of colleagues they would like alerted to their successful 
publication. If the product is published on CES4Health.info, the 
editor sends an e-mail letter to those individuals to highlight the 
author’s success, and to raise awareness of community-engaged 
scholarship among the author’s administrators and promotion and 
tenure committee members. 

Measuring the Impact of CES4Health.info 
in Year One 

Assessment of CES4Health.info is ongoing, and includes IRB- 
approved online surveys of authors, reviewers, and users (those 
who have downloaded one or more products). Key areas addressed 
with authors and reviewers include 

• satisfaction with the process of review—submission, 
communication, timeliness of response (for authors); 
identification as a reviewer, training, notifications, 
communication with editorial staff (for reviewers); 

• satisfaction with and value of the feedback received as 
a result of the review (for authors); 

• satisfaction with the guidance provided to reviewers to 
perform review (for reviewers); 

• ability to use the feedback from CES4Health.info in 
their individual tenure/promotion review process (for 
authors); 

• strengths of the process and opportunities for 
improvement; 

• utility of the CES4Health website; 

• value of the community-engaged scholarship criteria; 
and 

• other needs/resources for peer review that would be 
helpful. 
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Surveys of users focus on utility of the CES4Health.info 
website, perceptions of the products available for download, and 
recommendations for improvement of CES4Health.info. 

The Sample and Data Collection Process 

At the time of this analysis in November 2010,24 products had 
been peer-reviewed and disseminated through CES4Health.info, 
and 382 unique individuals had downloaded products. After the 
public launch of CES4Health.info, the corresponding authors of 
the 12 products accepted during the pilot phase and the reviewers 
of those products were sent an e-mail invitation with a link to an 
online survey. The corresponding authors and the reviewers of the 
12 products published after the public launch were sent an invi¬ 
tation e-mail with a link to an online survey approximately one 
month after publication of the product they authored or reviewed. 
User surveys were conducted in June, September, and November 
2010 in order to reach all users in CES4Health.info’s first year. The 
authors also collected statistics concerning user rates of accessing 
and downloading available products. There was possible overlap 
between author, reviewer, and user samples. Participants were sent 
separate requests and surveys for author, reviewer, and user data 
collection. Therefore the same individual might receive more than 
one survey request and complete these surveys separately. 

Data Analysis Process 

Survey responses to quantitative questions were counted and 
percentages calculated. Comments in response to qualitative ques¬ 
tions were reviewed to identify emerging themes and to choose 
descriptive quotations to illustrate points made below. 

Findings 

A total of 425 surveys were sent by e-mail to valid addresses. 
Response rates are indicated in Table 1. In this section, respondents 
are referred to as “authors,” “reviewers,” or “users,” reflecting their 
relationship with CES4Health.info. The overall response rate was 
26%; however, response rates for authors and reviewers were con¬ 
siderably higher than the response rate for users. Response rates for 
individual questions in the remainder of this section are variable, 
based on valid responses to each question (some respondents did 
not answer all questions). 



54 Journal of Higher Education Outreach and Engagement 


Table I. Survey Distribution and Responses 



Authors 

Reviewers 

Users 


N 

N 

N 

Number of surveys 
e-mailed 

24 

58 

343 

Number of respondents 

17 

36 

58 

Response rate 

71% 

62% 

17% 


Profile of the Respondents 

Ten of the responding reviewers and nine of the responding 
authors were also user respondents. Approximately 74% (n = 82) 
of respondents were employed in academic settings (Table 2). The 
affiliation of respondents not working in higher education varied 
greatly Almost one quarter (n = 7) each were from government 
agencies, from community-based nonprofits, and from hospitals 
or health systems. 


Table 2. Employed at a Higher Education Institution? 



Authors 

Reviewers 

Users 

Total 

Yes 

N = 13 

N = 23 

N = 46 

N = 82 


76% 

64% 

79% 

74% 

No 

N = 4 

N= 13 

N = 12 

N = 29 


24% 

36% 

21% 

26% 

Total respondents 

N = 17 

N = 36 

N = 58 

N = 1 1 1 


100% 

100% 

100% 

100% 


Author feedback. 

In general, the 17 responding authors were satisfied with the 
various elements of preparing for the submission of a product 
(Table 3). Where dissatisfaction was expressed, responses revealed 
that authors felt there was a lack of clarity in what can be submitted, 
and how to prepare and submit a product. Authors were most satis¬ 
fied with the responsiveness of editorial staff to their inquiries, with 
93% (n=14) indicating satisfaction. 
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Table 3. Author Satisfaction with Preparation for Submission 


Submission feature 

Strongly satisfied 
or satisfied 

Neutral 

Dissatisfied or 
strongly dissatisfied 

Communications about 

N = 12 

N = 2 

N = 1 

submitting a product for 
review 

80% 

13% 

7% 

Clarity of materials 

N = 1 1 

N = 3 

N = 1 

provided about the 
submission process 

73% 

20% 

7% 

Clarity of materials 

N = 12 

N = 2 

N = 0 

provided about the review 
process 

86% 

14% 

0% 

Clarity of review criteria 

N = 1 1 

N = 4 

N = 0 


73% 

27% 

0% 

Guidance for formatting a 

N = 9 

N = 5 

N = 1 

product for review 

60% 

33% 

7% 

Responsiveness of 

N = 14 

N = 1 

N = 0 

CES4Health.info staff to 
queries about submission 

93% 

7% 

0% 


Twelve (80%) authors indicated that they noted the product as 
a peer-reviewed publication on their curriculum vitae. Eight (72%) 
respondents felt that their CES4Health.info publication would 
make a difference in future performance reviews; three (27%) felt 
that it would not. One respondent stated: “As a peer reviewed pub¬ 
lication, it will definitely count on my tenure review.” 

Authors were asked what has resulted from their products 
publication. Five (about 50%) indicated that they had received rec¬ 
ognition from a supervisor or peers. One respondent commented 
that it “has provided a great way to disseminate the product on a 
national level.” 

Some comments offered by authors reflect the degree to which 
they recognize and appreciate the rigor of the process. For example, 
one wrote, “I appreciated the depth of background, justification, 
and rationale that was required of the reviewers. It instilled faith in 
the rigor and value of the peer review process.” 

Authors were asked their reasons for submitting a product for 
review and encouraged to choose multiple responses. The most fre¬ 
quent reason given was to get the product published (see Table 4). 
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Table 4. Reasons to Submit a Product for Review (Authors Only) 

Reason 

N 

% 

1 wanted to get published. 

10 

59% 

1 wanted to have it disseminated. 

9 

53% 

1 was curious to see what the review process 
would reveal. 

6 

35% 

1 am coming up for review and 1 wanted an 
external peer review. 

4 

24% 

1 wanted to get opinions on my work from indi¬ 
viduals outside of my organization. 

3 

18% 


Reviewer feedback. 

Overall, the 36 responding reviewers were satisfied with their 
preparation to be a reviewer (Table 5) and with the review process 
(Table 6). The only area of possible dissatisfaction was the time 
frame allowed to complete a review, which was 2 weeks at the 
beginning of the first year, but was later extended to 4 weeks. 


Table 5. Reviewer Satisfaction with Preparation for Conducting Reviews 


Preparation feature 

Strongly satisfied 
or satisfied 

Neutral 

Dissatisfied or 
strongly dissatisfied 

Communications about 

N = 34 

N = 0 

N = 0 

serving as a reviewer 

100% 

0% 

0% 

Timeframe between 

N = 32 

N = 1 

N= 1 

applying and being 
accepted as a reviewer 

94% 

3% 

3% 

Training to be a reviewer 

N = 30 

N = 4 

N = 0 


88% 

12% 

0% 

Clarity of materials 

N = 31 

N = 3 

N = 0 

provided for training 

91% 

9% 

0% 

Table 6. Reviewer Satisfaction with the Review Process 


Review feature 

Strongly satisfied 
or satisfied 

Neutral 

Dissatisfied or 
strongly dissatisfied 

Adequancy of timeframe 

N = 31 

N = 1 

N = 2 

to conduct review 

91% 

3% 

6% 

Clarity of review criteria 

N = 32 

N = 2 

N = 0 


94% 

6% 

0% 

Alignment of product 

N = 33 

N = 1 

N = 0 

topic with my expertise 

97% 

3% 

0% 

User-friendliness of online 

N = 28 

N = 5 

N= 1 

review form 

82% 

15% 

3% 
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User feedback. 

Between November 2009 and November 2010, there were 749 
product downloads of the 24 available products, with an average 
of almost two products per user, and with the number of down¬ 
loads per product ranging from 10 to 108. Overall, 81% of the 58 
responding users (n = 39) felt that the product they downloaded 
was very, or somewhat, useful. Users were asked for the reasons 
they decided to search CES4Health.info; they could select mul¬ 
tiple responses. Thirty (52%) indicated that they wanted to see this 
CCPH resource; 29 (50%) also indicated curiosity (Table 7). 


Table 7. Decision to Search CES4Health.info (Users Only) 


Reason 

N 

% 

1 find the CCPH resources useful and wanted to 
see this one. 

30 

52% 

1 was curious. 

29 

50% 

1 wanted to check it out before submitting a 
product for review. 

10 

17% 

1 wanted to check it out before offering to be a 
reviewer. 

9 

16% 

1 have used other similiar portals and wanted to 
try this one out. 

6 

10% 

1 was looking for a specific kind of product and 
one of my colleagues suggested 1 look at this 
website. 

3 

5% 


Users were generally satisfied with the CES4Health.info web¬ 
site and the information provided about available products. Only 
57% (n = 28) were satisfied with the number of products that 
matched what they were searching for; this likely reflects the early 
stages of development of this resource and the fact that the volume 
of products available is not yet very high. Ninety-five percent of 
user respondents (n - 45) expect to use CES4Health.info again 
in the future. Specific comments about future use included, “An 
easy-to-use website so it will be a regular stop for my work”; “I will 
be looking for examples of what types of work are published”; and 
“The more this develops, and more resources that are available, the 
more powerful this will become. I am excited for that!” 

Suggestions for Improvement 

Both users and reviewers were asked what would encourage 
them to submit a product to CES4Health.info in the future. 








58 Journal of Higher Education Outreach and Engagement 


The most frequent responses suggested that a web-based tuto¬ 
rial be created to guide authors through the submission process. 
Respondents also indicated the importance of increasing aware¬ 
ness about the types of products that would be appropriate for 
CES4Health.info. Users indicated that they would like to post 
reviews of products on the website as well as e-mail product authors 
directly from the site. 


Discussion 

The evaluation of CES4Health.info in its first year (November 

2009 to November 2010) pro¬ 
vides insight into the utility and 
potential impact of this resource. 
Authors see CES4Health.info 
as a venue for publication that 
offers them valuable feedback 
about their product, provides 
them with an opportunity to 
present their work as credible 
scholarship to their colleagues, 
and expands the reach of their 
community-engaged scholarship 
products. The authors believe 
that CES4Health.info addresses 
a common challenge—a lack 
of venue for peer review and 
publication of products of com¬ 
munity-engaged scholarship in 
forms other than journal arti¬ 
cles—and is, thus, valuable to 
faculty work and career advancement. Early anecdotal evidence 
suggests that CES4Health.info maybe helpful for recognizing com¬ 
munity-engaged scholarship in promotion and tenure processes. 

Limitations of the Assessment 

This first assessment of CES4Health.info has several limi¬ 
tations. First, the timing of the survey, only 1 year post-launch, 
limits the generalizability and scope of the findings. The sample size 
available at this early stage was small. Consequently, the number 
of respondents is too few to allow firm conclusions. Moreover, 
the authors cannot report on impact of CES4Health.info on fac¬ 
ulty authors’ promotion and tenure reviews, or the usefulness of 


“CES4Health. info 
addresses a common 
challenge—a lack of 
venue for peer review 
and publication 
of products of 
community-engaged 
scholarship informs 
other than journal 
articles—and is, thus, 
valuable to faculty 
work and career 
advancement.” 
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published products in such reviews, as most authors have not pur¬ 
sued promotion or tenure since their CES4Health.info publication. 
The timing of future evaluations will allow more authors to have 
completed career advancement processes and will focus on the 
impact of CES4Health.info on promotion and tenure. 

Second, the low response rate for users means that feedback 
was not secured from the majority of people who have downloaded 
products. Third, the sample included only those authors whose 
products were published and the reviewers who reviewed those 
successful products. These respondents were therefore not repre¬ 
sentative of all authors and reviewers who had participated. 

Next Steps for CES4Health.info 

The assessment of the first year of this endeavor suggests that, 
overall, authors, reviewers, and users see value in CES4Health.info. 
They are satisfied with its submission, review, and product search 
processes. Some areas noted by survey respondents as weaknesses 
have already been addressed. For example, some reviewers were 
concerned about the length of time permitted for review comple¬ 
tion. The review period was extended from 2 to 4 weeks during 
the first year. Other suggestions for improvement are being con¬ 
sidered. Continued assessment of CES4Health.info, and increased 
marketing and outreach, will help to enhance its use, usefulness, 
and impact. 

CES4Health.info will continue to be sponsored by Community 
Campus Partnerships for Health with editorial offices provided 
by the University of Minnesota Extension’s Children, Youth and 
Family Consortium, the affiliation of the founding editor. Funding 
is being sought to make improvements to the online system and to 
support themed calls for products, collaborations with other pub¬ 
lications as well as community partners, and initiatives to support 
the application of published products in communities. 

Conclusion 

CES4Health.info provides individuals working to improve 
health in communities with accessible, useful information typically 
not published in journals by disseminating an array of products 
that have been reviewed and deemed high quality by community 
and academic peers. Recognizing the lack of peer-reviewed pub¬ 
lication outlets (Nyden, 2003) and the absence of community voice 
in the peer review of community-engaged work (Ellison & Eatman, 
2008), the Commission on Community-Engaged Scholarship in the 
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Health Professions (2005) called for the development of a national 
board to provide peer preview of innovative products of commu¬ 
nity-engaged scholarship. CES4health.info answers that call by 
providing a mechanism for the peer review and online publication 
of innovative scholarly products, increasing the chances that these 
products will be “counted” in promotion and tenure decisions (ASA 
Task Force 2005; Ellison & Eatman, 2008; Gelmon & Agre-Kippenhan, 
2002; Jackson, Schwartz, & Andree, 2008; O’Meara & Edgerton, 2005). 
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